1. Field
The present invention relates to video technology, and more specifically to video image stabilization.
2. Background
The integration of camera and video functionality into mobile phones, personal digital assistants (PDAs) and other handheld devices has become mainstream in today's consumer electronic marketplace. This present capability to add imaging circuits to these handheld devices is attributable, in part, to the availability of advanced compression techniques such as MPEG-4. Using MPEG or another appropriate compression scheme, video clips can be taken by the camera and transmitted wirelessly to other devices.
The transmission of the video may take place in real time or non-real time. Video e-mail is one increasingly popular non-real time technique used in several markets around the world. Using video e-mail, an individual can use a handheld device to take a video or multimedia clip, compress and record the video, and then transmit a compressed version of that clip together with an appended audio or video message to another computing device (such as a PC or another handheld device). The receiving device, in turn, can record the clip, or decode and reproduce the clip on a display. Real time imaging techniques are also in development. As processors become faster, compression techniques superior, and wireless bandwidths larger, video telephony in real time using handheld devices will likely enjoy an ever-increasing marketplace presence.
Camera or image shake represents an issue characteristic of any video imaging system. Camera shake is unintended movement of the video camera device by the user that, if not compensated for, appears in the rendered image sequence. Camera shake often includes small, jerky, and alternating types of movements. Camera shake should be distinguished from normal, intended motion of the camera associated with scene scanning by a videographer. More often than not, camera shake contributes nothing to the rendering of a scene. Instead, the shake can compromise the quality of the video and, more than anything, is annoying to the viewer. While the problem is universally applicable to free-moving video cameras, the adverse effects associated with image shake are only exacerbated in lighter and smaller devices such as mobile phones or the like.
Certain techniques have been proposed or implemented to reduce or eliminate camera shake. For example, image stabilization circuit techniques are often used in camcorders to help remove unwanted “shake” associated with jerky and unintended movements of the camera by the individual shooting the image sequence. Conventional techniques involve studying the motion of the video frames relative to one another, quantifying this motion using frame-level motion vectors, integrating these vectors together, enlarging and cropping the image, and using the integrated vector information to “reposition” frames of the image to produce a more smoother image sequence.
For a growing body of applications, the present method is no longer adequate. The need for more effective yet inexpensive image stabilization techniques has dramatically increased by factors such as (i) the marketplace demand for smaller and smaller cameras and video recorders, and (ii) the incorporation into various handheld devices (e.g., mobile phones, personal digital assistants, GPS receivers, etc.) of camera and video functionality. As the electronic devices become smaller in form factor, they unfortunately permit grip capability that is less “user friendly” than more conventional or specially-designed grips such as those found on heavier or more stable models, or on over-the-shoulder video cameras. Further, as handheld devices become lighter, it is more difficult for the user to shoot images that are free of discernable shaking of the hand or other unintended user movements, which movements become incorporated into the image. Additionally, small handheld devices have little room for the incorporation of additional cumbersome circuitry dedicated exclusively to image stabilization. Cost also becomes an important issue in these handheld devices.
The traditional methods have shortcomings, particularly when proposed for use in handheld devices. One problem with the traditional method used in camcorder applications is that it often cannot distinguish natural, intended motion associated with scene scanning or moving objects on one hand, from undesirable and unintended motion associated with camera shake on the other hand. As a result, the device may attempt to compensate for motion that is a natural and desired part of the scene being recorded, resulting in inaccuracies and visually unpleasant artifacts at the output. As noted above, the problem is exacerbated in the case of a light weight hand held device, where unstable motion or shake is typically more discernable.
As a consequence of the various shortcomings that persist with conventional stabilization systems, the rendered video image can be significantly compromised or even corrupt. For example, image shake tends to induce motion blur in the final rendered image sequence even when the image stabilization function is engaged. Further, if the area being scanned lacks sufficient contrasting detail for the stabilizer to “lock onto”, the stabilizer can hunt, oscillate or bounce. These errors are only magnified in the output video sequence. Another common problem is that, as mentioned above, the stabilization system often cannot discern whether the movement of the object or camera is natural, intended movement, or camera shake. In this case, when the user commences a slow pan or tilt, the stabilizer may erroneously predict the commencement of this movement to be camera shake and proceed to compensate for it. The result is an unstable or inaccurate output image sequence.
Accordingly, a need exists in the art to remove unstable motions in video handheld and other devices while preserving natural motion such as scene scanning, with a minimal requirement of additional dedicated circuitry and a minimal increase in computational complexity.